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STRENGTHENING ASSESSMENTS OF SCHOOL CLIMATE 


Strengthening Assessments of School 
Climate: Lessons from the NYC School 

Survey 

More and more cities and states are using surveys to collect information about 
school climate from students, teachers and parents. These surveys have the potential 
to shed light on critical aspects of the learning environment, and they are being 
incorporated into a growing number of public and privately funded education 
initiatives. The US Department of Education’s Safe and Supportive School grants, 
for example, support survey efforts in 11 states, which are gathering input about 
student engagement and safety, among other issues. ’ 

The Research Alliance for New York City Schools has been working with the NYC 
Department of Education (DOE) to review and enhance its School Survey — the 
largest in the nation — since 2010. This work has resulted in several concrete 
improvements to the NYC School Survey, and has pointed to a variety of lessons for 
other cities that are engaged in similar efforts. This brief summarizes our findings to 
date (which are explored in more depth in a technical paper available on our 
website^), and presents both the Research Alliance’s and the DOE’s reflections 
about the process of improving the School Survey. It outlines the Research 
Alliance’s specific recommendations for changes to the NYC School Survey and next 
steps for analysis and improvement, as well as a set of broader lessons that have 
emerged from our work. For more information about our ongoing research on the 
NYC School Survey, please visit our website: www . rany cs . org . 


About the NYC School Survey 

Each spring, the DOE invites all public school students in grades 6 through 12, as 
well as parents and teachers throughout the City to complete the School Survey. In 
2012, 476,567 parents, 428,327 students, and 62,115 teachers completed the NYC 
School Survey. The survey is designed to elicit input about the environment at each 
school, including Academic Expectations, Communication, Engagement, and Safety 
& Respect, which together constitute the School Environment score on the Progress 
Report. School Survey information contibutes 10-15 percent (depending on school 
type) to each school’s annual Progress Report grade. It is also intended to “support a 
dialogue among all members of the school community about how to make the 
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school a better place to learn” and to help 
school leaders “better understand their own 
school's strengths and target areas for 
improvement.”^ 

When the DOE began to develop the School 
Survey in 2006, it collaborated with a range of 
stakeholders in NYC public education, 
including researchers, community-based 
organizations, educators, parents, and internal 
DOE staff. It built on the survey development 
work of the University of Chicago 
Consortium on Chicago School Research 
(CCSR — see “Research-Practice Partnerships” 
textbox on page 5 for more information) . The 
DOE incorporated some items from CCSR 
surveys and from other well-established 
surveys and also created items specific to NYC 
schools. 

The NYC School Survey represents a major investment of resources and time for 
schools and the district as a whole. School Survey scores, combined with 
attendance, are the only non-academic indicators used in the City’s Progress 
Reports, which in turn are the primary way the DOE evaluates overall school 
quality. It is also the sole source of community feedback in the Progress Report. 

Given the high stakes nature of the survey, establishing the reliability and validity of 
the measures is critical. Reliability describes the extent to which the measures 
remain consistent under varying conditions — for example, do two parents from 
different schools who feel the same about the quality of the school environment 
respond to the survey items similarly? In contrast, validity refers to how well a 
measure represents the intended idea — for example, does a school with a high score 
on the School Survey actually have a high-quality learning environment? 

Rockoff and Speroni conducted a preliminary assessment of the reliability and 
validity of the 2008 School Survey. This initial look found high levels of reliability 
for the four categories reported on Progress Reports and mixed validity evidence."^ 


The NYC Survey's Census 
Approach 

A common approach to survey research is 
to sample from the total population, 
randomly selecting a subgroup of survey 
respondents from the total pool of 
respondents. However, the NYC DOE uses 
a census approach that asks all parents, 
students in grades 6-12, and teachers to 
complete the survey. NYC public schools 
employ over 80,000 teachers and serve 
more than 1.1 million students (and by 
extension, their parents). This makes the 
NYC School Survey one of the largest 
surveys in the country, second in size only 
to the US census. The DOE uses the 
approach to fulfill its goals of engaging the 
whole school community and soliciting 
feedback from all constituents. 



4 


STRENGTHENING ASSESSMENTS OF SCHOOL CLIMATE 


DOE officials were interested in continuing to develop and improve the survey, but 
due to the firewall protecting respondents’ confidentiality, they did not have access 
to the individual-level data required for such analysis. The Research Alliance was a 
natural partner, because of our unique position to access the information needed — 
and the capacity to analyze data from such a large-scale survey. In 2010, the 
Research Alliance began work on the NYC School Survey, with the goal of 
examining the reliability and validity of the measures over time. 

Using data from the 2008, 2009 and 2010 survey administrations,^ we undertook a 
systematic examination designed to assess how well the School Survey was capturing 
key stakeholders’ experiences and to inform potential improvements to the survey. 
We worked closely with the DOE to understand the survey’s context and goals and 
ultimately made a number of recommendations about how the survey could be 
improved. As outlined in the “Next Steps” textbox on page 8, we are continuing to 
work with the DOE to make the survey more effective. The major findings to date 
are summarized below. 

Findings About the NYC Schooi Survey 

Our analysis of three years of School Survey data revealed several important findings 
about the generalizability of survey results and the reliability and validity of the 
survey measures: 

Response Rates. Robust response rates for students and teachers demonstrate 
widespread participation, a key strength of the NYC School Survey. Response rates 
among students and teachers increased steadily over time and reached 78 percent 
and 83 percent, respectively, in 2010. These high response rates offer confidence 
that survey results reflect the opinions of the broader population. Parent response 
rates (49 percent in 2010 and 53 percent in 2012) did not approach the same levels 
as student and teacher response rates, and thus the representativeness of parent 
survey results is more in question. Elowever, it is important to consider that, 
historically, response rates for parent surveys in large school districts have been low 
(an estimated 30 percent for similar district-sponsored surveys^). By comparison, 
the parent response rate in NYC is high. The district has made it a priority to 
increase parent response rates, which have risen steadily over time. These positive 
trends in parent response rates are encouraging. 
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Reporting Categories. The NYC School Survey is complex in nature. The survey 
includes items that map to four predefined reporting categories (Academic 
Expectations, Communication, Engagement, and Safety & Respect) for three 
different reporters (parents, students, and teachers). Therefore, we examined the 
School Survey from many angles — deconstructing responses by reporting 
categories, respondents, and individuals within schools — to fully understand each 
component. Through this process, we replicated DOE’s four reporting categories 
and evaluated their ability to empirically measure different aspects of a school’s 
learning environment. In this work we were primarily interested in examining 
whether a construct like Academic Expectations was statistically distinct from 
another construct, such as Engagement, or if the two reporting categories were 
actually capturing a larger global perception of the school environment. 


Research-Practice Partnerships 

Research-practice partnerships are designed to better position research to contribute to improving 
outcomes for students and schoois by forming ongoing, coiiaborative reiationships between 
researchers and schooi districts.* These partnerships come in different shapes and sizes, depending 
on the needs of the schooi district and the capacity and interests of the invoived research organization. 
Since 2008, the Research Aiiiance for New York City Schoois has worked with the NYC DOE to 
conduct rigorous research on topics that matter to the City’s pubiic schoois. Our roie in anaiyzing the 
NYC Schooi Survey offers one exampie of the unique and important work that is possibie through 
research-practice partnerships. 

The Research Aiiiance was modeied after the University of Chicago Consortium on Chigaco Schooi 
Research (CCSR), which was founded in 1990 as a partnership between researchers from the 
University of Chicago, officiais from Chicago Pubiic Schoois, and other stakehoiders in Chicago. One 
of CCSR’s key contributions to the fieid has been the My Voice/My School Surveys, administered 
bienniaiiy to principais, students, and teachers in Chicago Pubiic Schoois. The surveys have gathered 
invaiuabie information about Chicago’s schoois, and the resuiting data and anaiyses have become 
infiuentiai in iocai and nationai schooi improvement and reform efforts. The DOE buiit on CCSR’s work 
as it deveioped a Schooi Survey for New York City. 

in recent years, CCSR and the Research Aiiiance for New York City Schoois have been joined by a 
growing number of research-practice partnerships in other cities, inciuding the Baitimore Education 
Research Consortium and the Newark Schoois Research Coiiaborative. These coiiaborations are 
reiativeiy new. Moving forward, we wiii continue to document iessons and insights emerging from our 
work — and from our unique roie as both a partner and evaiuator of iocai schooi district efforts. 

* See Coburn et ai., 2013. 
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Our analysis revealed three key findings: 

• The existing School Survey items did provide statistically reliable indicators of 
the doe’s four reporting categories. In other words, responses to the individual 
survey items used to create each of these measures were highly correlated with 
one another. 

• However, the four reporting category measures were statistically 
indistinguishable from each other. For the purposes of the school Progress 
Reports, this means that the survey items used to create the individual measures 
could be combined into a single “school environment” measure without 
significantly diminishing the current breadth of information about schools. 

• Because many of the items were so highly correlated with one another, the 
single global measure of the school environment could be constructed reliably 
using about half of the items currently being used to construct the four 
individual measures. 

Reporters. Although the reporting categories were not empirically different from 
one another, the three respondent groups were distinct. This suggests that the 
School Survey tapped into parents’, students’, and teachers’ unique perspectives 
about the school environment. In fact, a single school environment score for each 
respondent group may provide richer information about a school than four 
reporting category scores that combine respondents’ answers. 

Distinguishing Between Schools. Overall, the four reporting category scores were 
not strong measures for distinguishing among schools. In other words, the survey 
provides more information about differences between individuals within a school, 
and less information about how that school differs from other schools. When we 
examined reporting category scores bj respondent, we found that teachers’ responses 
were best able to distinguish between schools. Taken together, these results suggest 
that School Survey scores could be combined in ways that take into account the 
unique perspectives of each respondent group. For example, weighting teachers’ 
scores more heavily in the overall School Environment score on the Progress Report 
may be prudent, given that teachers’ reporting category scores more reliably 
distinguish between schools than parent and student scores. 
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Associations with School Outcomes. School Survey scores were significantly 
associated with other policy-relevant school characteristics, including student test 
scores and graduation rates. However, these associations were not consistent across 
years and reporters. Inconsistent associations over time add to the difficulty of 
summarizing how the School Survey relates to 
other school characteristics. When there were 
significant associations, relatively large 
differences in School Survey scores were 
associated with relatively small differences in 
test scores. For high school outcomes, 
differences in School Survey scores were 
associated with small but meaningful 
differences in the percent of students “on track” 
to graduate. This suggests that by improving 
aspects of the school environment in high 
schools, there is the potential to increase the 
percent of students who are on track and 
ultimately graduate. 

Recommendations 

The universal administration of the NYC 
School Survey and its corresponding high 
response rates present an opportunity to accomplish a wide range of goals that the 
DOE and other school systems increasingly see as a priority. These goals include 
giving parents, students, and teachers a voice in assessing the quality of their 
schools, providing information that can be used for school improvement efforts, and 
constructing climate measures for which schools may be held accoimtable. Findings 
from the Research Alliance’s analyses suggest, however, that some of this potential 
remains untapped in NYC. The Research Alliance has made the following 
recommendations to the DOE, which are intended to increase the utility of the 
School Survey and incrementally improve the measures. See the “Policymaker 
Perspective” section for information about how these recommendations have been 
utilized. 

• Eliminate redundant items that are used to create the school environment 
measures, while preserving the reliability of those measures. 


Next Steps 

The Research Alliance for NYC 
Schools and the DOE are engaged in 
an ongoing collaboration to further 
develop the parent, student, and 
teacher School Surveys. In the coming 
months, we will continue the cycle of 
analysis, revisions, and reanalysis with 
the shared goal of improving the 
quality of information available about 
NYC’s schools. For example, the 
Research Alliance is working to 
validate new measures from the 
teacher survey. In addition, the DOE is 
continuing to consider revisions to the 
way School Surveys are scored and 
reported for the Progress Report. 
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• Reduce the time required to complete the School Survey and improve the 
reliability of measures by creating more consistent question formats and 
response categories. 

• Choose different and separate measures for the parent, student, and teacher 
surveys to capitalize on their distinctive perspectives on their schools. Limit or 
eliminate measures that combine responses from parent, student, and teacher 
surveys. For example, although teachers may be the best reporters of Academic 
Expectations, parents and students may have imique and important views on 
Safety & Respect. 

• Incorporate new school environment measures that are more likely to 
distinguish between schools and are associated with other school performance 
indicators. For example, ask teachers to report about their principal’s 
instructional leadership. 

• Incorporate measures that more effectively gauge parent satisfaction and 
engagement with their child’s school. For example, items like “I would 
recommend this school to other parents” tap into parents’ overall satisfaction. 

Lessons for Other School Survey Efforts 

In addition to the specific recommendations outlined above, our work also suggests 

a number of larger lessons for other cities that are developing school surveys. 

• Researchers should be brought into the process early. Incorporating 
the best thinking on survey development — including expertise in rating-scale 
construction, diverse survey respondents (e.g., adolescents, people from 
different cultural backgrounds, low literacy), sampling frame, scoring 
procedures, and school-level measurement across elementary, middle, and high 
school — is important to create a survey that accurately reflects people’s 
experiences in schools. Having a long-term research partner can help ensure 
that surveys use rigorous, well-tested measures and methods and that they 
address areas established as important in previous research about school climate. 

• A continuous improvement approach to school surveys is critical. 

States and local school districts should examine survey results and 


The Research Alliance for 
New York City Schools 


9 


implementation experiences and work to make surveys better with each 
iteration. Although this will likely pose a challenge to examining measures over 
time, it is helpful to engage in an ongoing survey improvement process where 
small incremental changes are made and pilot tested on a yearly basis. Based on 
results of pilot tests, future surveys can integrate new questions and possibly 
eliminate redundant items. 

• Measures of the school environment should be given more weight in 
accountability efforts. The NYC School Survey results contribute a small 
portion to the annual Progress Report grade issued to every school. The CCSR 
surveys are not currently used for accountability purposes, although there has 
been some consideration of doing so.’ If good measures of school climate are 
used, they represent an important aspect of educators’ work and effectiveness. 
There is good and growing evidence that a school’s environment, including 
safety and student and family engagement, is important for improving academic 
outcomes.^ 

• Policymakers at multiple levels should consider using their school 
survey to collect additional kinds of data that could help assess 
student needs and progress. For example, nonacademic outcomes like 
motivation, academic aspirations, and problem solving are increasingly viewed 
as important to students’ success in school and in life. Schools surveys could 
incorporate questions about these kinds of student outcomes, offering valuable 
information about how schools may be influencing these important nonacademic 
domains. 

• Finally, surveys might be used to collect information on teacher 
quality from students. Ongoing research has shown that students’ ratings 
tend to track closely with other ways of evaluating teachers, such as student test 
score gains, direct classroom observations, and videotaped observations of 
teacher practice.^ It may make sense to integrate student ratings, collected 
through a school survey, as a small part of teacher evaluation systems. 

Surely, continuing work in NYC and around the country will generate additional 
lessons in years to come. In the meantime, it is encouraging that so many districts 
have begun using surveys to engage members of the school community. Capturing 
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the perspectives of parents, students, and teachers is vital for evaluating and 
improving the nation’s schools. 
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Policymaker Perspective 

As the largest annual survey in the country, the NYC School Survey is a powerful 
tool that helps the NYC Department of Education (DOE) and school communities 
learn what parents, teachers, and students in grades 6-12 think about their school’s 
learning environment. Survey results are published on the DOE’s website, and each 
school is issued a survey report that school leaders, students, and families can use to 
facilitate a dialogue about the school environment and how to make the school a 
better place to learn. Importantly, because School Survey results account for 10 to 
15 percent of a school’s Progress Report grade, the DOE must balance the interests 
of obtaining actionable data for schools and differentiating data that can be used as 
part of our accountability system. 

On an operational level, our partnership with the Research Alliance has been critical 
to implementing the School Survey. Due to the confidential nature of the parent and 
student surveys, the DOE cannot analyze respondent level data. However, as an 
independent entity, the Research Alliance is able to conduct these analyses and share 
the results with the DOE while preserving the School Survey’s confidentiality. 

In addition, the Research Alliance’s years of studying our education system has 
resulted in a deep understanding of the complex issues facing our schools. This 
background helps inform the Research Alliance’s perspective regarding the School 
Survey’s content, scoring, and use. Further, the Research Alliance synthesizes 
survey data, research, and expert analyses from around the country that the DOE 
uses as part of its annual survey review process. All of this information helps us 
understand the utility of existing survey questions and the impact of changing, 
adding, and removing questions. 

In response to the Research Alliance’s feedback and our own assessment of how 
certain changes would impact schools and other stakeholders who use survey results 
to help inform program-related decisions, the DOE has made a number of 
significant changes to the School Survey. For example, last year we piloted a set of 
test questions on the teacher survey that were recommended by the Research 
Alliance. Based on the Research Alliance’s analysis of these questions, we have 
incorporated a number of them into the 2012-13 teacher survey with the goal of 
obtaining data that better measures school performance indicators. We also revised 
the 2012-13 parent and student surveys based on the Research Alliance’s 
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recommendations by removing redundant items and adding new items. For 
example, we added improved questions about engagement and satisfaction levels to 
the parent survey, as well as a set of non-scored questions about self-efficacy to the 
student survey. We also streamlined the format and structure of all three surveys in 
an effort to make the surveys more accessible and user-friendly. 

The Research Alliance brings a strong academic lens to the analysis of survey results. 
Flowever, as a school district, we must consider a number of factors in designing the 
survey in order to get actionable information that can be used for school 
accountability. Most fundamentally, survey questions need to focus on conditions 
that are within the control of the school and that measure the school as a whole — a 
perspective shared by the Research Alliance. Further, we seek and incorporate 
stakeholder feedback into the survey revision process to ensure our accountability 
system produces an accurate picture of school quality. For example, we engage a 
range of DOE programs and teams to learn how they use the survey. Teacher and 
parent advocacy groups also provide feedback about survey content that is 
important to them. These sources of feedback are not always consistent with one 
another, and it is not possible to accommodate every request in a brief survey. 
Similarly, with respect to the Research Alliance’s recommendation to shift to a 
respondent-based scoring model, the DOE must consider how such a change would 
affect the way school commimities understand, interpret, and use survey results. 
These are just some examples of the diverse set of interests and concerns the DOE 
must balance as a part of an ongoing survey revision process. 

In addition, our objective is not solely to produce survey results that are correlated 
with other school performance indicators, such as student test scores and graduation 
rates. We have purposefully incorporated diverse measures of school quality into 
our accountability system — namely, the Progress Report, Quality Review, and 
School Survey — with the goal of capturing a broad and rich picture of schools’ 
strengths and weaknesses. We believe that parents’ evaluations of schools draw on 
aspects of school quality beyond the information we capture through test scores, 
classroom observations, and other data streams. Since no one score or grade can 
fully reflect every nuance of school performance, these accountability tools are 
designed to provide families and school communities with multiple, and varied, data 
points on a school. We work hard to ensure that families understand these data 
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points and use them to make informed decisions about where to send their child to 
school. 

The DOE will continue to refine all three versions of the School Survey in light of 
the Research Alliance’s subsequent analyses and recommendations. In addition, we 
are currently reviewing the Research Alliance’s recommendations regarding the 
scoring of the School Survey results for accountability purposes. We look forward 
to continued collaboration with the Research Alliance as we work to strengthen one 
of our most valuable feedback and accountability tools. 
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Notes 

' See the Safe and Supportive Schools 
website: 

http:/ / safesupportiveschools.ed.gov/index 
■ |ihp?id— 1 33 . See also Shah (2013). 

^ Nathanson et al. (2013). 

'NYC DOE (2013). 

'' Rockoff & Speroni (2008). 

' Because the Survey underwent significant 
changes during the first two years of 
administration, we decided to make 2008 — 
the first year that the survey exhibited 
continuity with future administrative 
years — the base year for our analysis. 

" We drew several comparisons to place NYC 
School Survey parent response rates in 
context. Austin (201 1) estimated that 
typical parent response rates for district- 
sponsored paper surveys is 30 percent and 
noted that NYC DOE made a heavy 
investment in public relations outreach to 
get parent response rates to 45 percent in 
the 2009 School Survey administration. 


Other examples suggest even lower rates: A 
Cincinnati Public Schools news release 
reported an 8 percent parent response rate 
in 2012. And, based on a 2008 report about 
a Los Angeles Unified School District parent 
survey, we estimated that 22 percent of the 
1 5 ,000 parents who received a survey 
responded (the response rate was not 
directly reported in the paper, and we based 
our estimate on Tables 1 and 2, which 
reported the number of responses to two 
specific survey items.) 

Easton et al. (2008). 

* Center for Social and Emotional Education 
(2010); Cohen et al. (2009); Zins et al. 
(2004). 

^ Camburn (2012). 
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